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SURRENDER THE ORIGINAL PATENT, AND ASSENT OF ASSIGNEE TO REISSUE 

HONORABLE ASSISTANT COMMISSIONER FOR PATENTS, 
Washington, D.C. 20231 

SIR: 

In accordance with 35 U.S.C. § 251, Applicant is hereby filing a reissue 
application with claims that are broader than those issued in the original patent. 

Please enter the following amendment prior to examination so that there is a 
total of seven (7) independent and a total of forty-five (45) claims pending in the 
application. 
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AMENDMENT 

Amendment to Claims 

Please add the following claims as shown below. 

23. (Newly added) An apparatus adapted to process an image. comDrisina: 
an amplifier adapted to apply more than one gain level to the image: and 

a processor coupled to the amplifier, wherein the processor is capable of 
adiustinq the more than one gain leyel applied by the amplifier, 

24. (Newly added) The apparatus of claim 23, further comprising a sensor 
coupled to the amplifier provide the amplifier with the image, 

25. (Newly a dded) The apparatus of claim 23, wherein the processor is 
further capable of p roviding a control signal to the amplifier to adjust the more than 
one gain level applied by the amplifier, 

26. (Newly a dded) The apparatus of claim 23, wherein the amplifier is 
capable o f applying different gain levels to different regions of the imaoe, 

27. (Newly a dded) The apparatus of claim 23, wherein the processor is 
capable of updating a gain map comprising settings applied bv the amplifier. 

28. (Newly added) The apparatus of claim 27, wherein the gain map is a two 
dimensional array of g ain levels, each gain level indicating a particular gain applied 
by the amplifier to a corresponding region of the image. 

29. (Newly ad ded) The apparatus of claim 27, further comprising a register 
coupled to the processor and the amplifier. 

30. (Newly ad ded) The apparatus of claim 29, wherein the register is capable 
of storing the gain m ap and the amplifier is adapted to read the gain levels from the 
register. 
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31 ■ (Newly a dded) The apparatus of claim 25, wherein the processor is 
further adapted to provide the control signal to the amplifier in real-time. 

32- (Newly a dded) The apparatus of claim 24, wherein the processor is 
further adapted to analyze the image to determine if the sensor is providing a 
sufficient level of detail. 

33. (Newly a dded) The apparatus of claim 32, wherein the processor is 
further adapted to i ncrease the gain level in dark portions of the image and 
decrease the gain level in bright portions of the image. 

34. (Newly added) An apparatus capable of processing an image, comprising: 
an image cap ture device coupled to the apparatus and capable of providing 

the image: 

an amplifier coupled to the camera, wherein the amplifier is adapted to apply 
at least t wo gain levels, each to a different region of the image: and 

a processor coupled to the amplifier, wherein the processor is adapted to 
provide a signal to the amplifier to adiust the at least two gain levels. 

35. (Newly a dded) The apparatus of claim 34, wherein the imaoe capture 
device is a camera. 

36. (Newly ad ded) The apparatus of claim 34. wherein the processor is 
capable of generatin g a gain map containing gain levels applied by the amplifier. 

37. (Newly ad ded) The apparatus of claim 36. wherein the gain map is a two 
dimensional array of gain levels, each gain level indicating a particular gain applied 
by the amplifier to a region of the image. 

38. (Newly added) The apparatus of claim 37, wherein the processor is 
capable of dividing the image into a two dimensional array of image regions, each 
image re gion being associated with a corresponding level in the gain map. 
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39. (Newly a dded) The apparatus of claim 34, further comprising a register 
coupled to the processor. 

40. (Newly a dded) The apparatus of claim 39, wherein the register is capable 
of storing a gain map and the amplifier is capable of reading the gain levels from 
the register. 

41 ■ (Newly added) A method of processing an image, comprising: 
amplifying a first portion of the image to a first gain level: 
amplifying a second portion of the image to a second gain level: and 
updating the first gain level and the second gain level in a gain map. 

42. (Newl y added) The method of claim 41 , wherein updating the first gain 
level and the second gain level is performed in response to clipping of a portion of 
the image. 

43. (Newly added) The method of claim 41 , wherein updating the first gain 
level and second gain level includes increasing the gain level in dark portions of the 
image and reducing the gain level in bright portions of the image, 

44. (Newly added) The method of claim 41 , wherein updating the first gain 
level and the second gain level includes dividing the image into a plurality of image 
regions, wherein each of the plurality of image regions is associated with a gain 
level in the gain map. 

45. (Newly added) The method of claim 41 > further comprising analyzing 
each ima ge region and updating the associated gain level in response to clipping of 
the image. 
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Remarks 

Applicant hereby requests that the United States Patent and Trademark Office 
initiate broadening reissue proceedings for the above-mentioned issued patent. 
Applicant requests that the preceding claims be added prior to examination. 

Specification 

A copy of the original patent is filed concurrently with this application. The 
copy of the original patent was prepared by scanning a copy of the original 
application as it was filed on December 2, 1 996. In addition, the text of the 
enclosed application has been modified, as required by 37 CFR § 1.173, to reflect 
the changes made by the preliminary amendment filed on December 2, 1996, the 
amendment filed on June 10, 1997, and the Certificate of Correction issued on 
June 9, 1998. These amendments have been shown as normal text as they appear 
in the original patent. No new matter has been added. 

Oath and Declaration 
Applicant has filed concurrently with this reissue application an oath and 
declaration that is believed to meet the requirements of 37 CFR §1 ,1 75, 

Offer to Surrender the Original Patent 
Patent No. 5,712,682 granted to Eric C. Hannah on January 27, 1998 of 
which Intel Corporation, now sole owner by assignment, and on whose behalf and 
with whose assent the accompanying application is made, hereby offers to 
surrender said letter patent. 

Assent of Assignee to Reissue 
The undersigned assignee of the entire interest in the above-mentioned letters 
patent hereby assents to the accompanying application. 

Drawings for the Reissue Application 
In accordance with M.P.E,P. §1413, Applicant hereby requests transfer of the 
drawings from the file of the original patent in lieu of new drawings. Applicant is 
enclosing with this reissue application a photocopy of the drawings of the patent as 
temporary drawings as permitted by 37 CFR § 1 .1 74, 
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Request for Abstract of Title Report 

Applicant is filing herewith a request for a certified Abstract of Title Report 
and appropriate fee in accordance with 37 CFR § 1.171. 



As indicated above^ claims 23-45 have been added. Support for the 
amendments is at least shown by the examples in FIG. 2-3 and described in 
Applicant's specification. Support for the amendments can also be found in claims 
1 -22 of the original patent. 

Applicant respectfully submits that no new matter has been added. 



Applicant respectfully requests that the Examiner allow all pending claims. 

Should It be determined that an additional fee is due under 37 CFR §§U 6 or 
1.17^ or any excess fee has been received, please charge that fee or credit the 
amount of overcharge to deposit account #02-2666. 

If the Examiner believes that there are any informalities which can be 
corrected by an Examiner's amendment, a telephone call to the undersigned at 
(480) 554-9732 is respectfully solicited. 



Support for Amendments, 



Conclusion 



Respectfully submitted, 
Eric Hannah 




Kenneth M. Seddon 
Patent Attorney 
Reg- No. 43,105 




c/o Blakely, Sokoloff , Taylor & Zafman, LLP 
12400 Wilshire Blvd., Seventh Floor 
Los Angeles, CA 90025-1026 
(503) 264-0967 



6 



TOTAL P,07 



042390.P3294CR 



APPLICATION FOR UNITED STATES LETTERS PATENT 



FOR 



CAMERA HAVING AN ADAPTIVE GAIN CONTROL 



Inventor(s): Eric C. Hannah 



Prepared by: Kenneth M. Seddon, 

Intellectual Property Attorney 

Intel® 

Intel Corporation 
Legal Department 
5000 W. Chandler Blvd. 
Chandler, AZ 85226-3699 



"Express Mail" label number EL414969674US 



1 



BACKGROUND OF THE INVENTION 
Field of the Invention 

The present Invention relates to Image processing. More specifically, a system for 
capturing and processing video images. 

Background 

Digital cameras are used in a variety of applications requiring image capture and image 
processing. Many applications require cameras which are economical, yet generate a high 
quality video signal. Typical solid state sensors used in digital cameras have a dynamic range (or 

ftpht intensity range) of 1000:1 or greater. However, many existing digital cameras utilize an 

'43 

ifflexpensive 8 bit or 6 bit analog-to-digital (A/D) converter to generate a digital output signal 
^presenting the captured Image. The use of an 8 bit A/D converter limits the dynamic range of 




ft' 

Iftjtput signal by providing a maximum of 256 possible luminance levels. Therefore, the dynamic 
lange capabilities of the sensor (1000:1) are compressed to 256:1 by the A/D converter. This 
pduction in dynamic range results in "dipping" of the image; i.e., loss of image detail in bright 
areas and dark areas of the image. When clipping occurs, dark areas of the image become black 
(e.g., luminance level 0) and bright areas of the image become white (e.g., luminance level 
255). 

Existing cameras attempt to compensate for this dynamic range reduction by using an 
automatic gain control (AGO amplifier having different gain settings. The gain setting of the 
AGO amplifier is determined based on the total luminance entering the camera. Since a single 
luminance level is determined, the selected gain setting is applied uniformly to the entire image. 
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An example of an existing camera is illustrated in Figure 1 . A digital camera 10 is attached to an 
image processor 12. Camera 10 captures an image of scene 14 and generates a digital signal 
representing the captured image, A solid state sensor 1 6 in camera 10 captures an image of 
scene 14. Typical solid state camera sensors are capable of discriminating light intensity over a 
dynamic range of 1000:1 or greater. Sensor 1 6 generates a sensor output signal representing 
the sensed image and provides the signal to an AGC amplifier 18. AGC amplifier 18 applies a 
particular gain setting to the sensor output signal. A/D converter 20 receives an output signal 
from AGC amplifier 18. As discussed above, A/D converter 20 may be an 8 bit or 6 bit 
converter. Using an 8 bit A/D converter 20, the dynamic range of the sensor output signal is 
0)mpressed to 256:1 . Similarly, using a 6 bit A/D converter 20, the dynamic range of the 
#nsor output signal is compressed to 64:1 . This dynamic range compression results in a loss of 
page detail at both ends of the dynamic range; i.e., bright areas and dark areas. As a result, 
^fight areas of the image are compressed into all white regions and dark areas of the image are 
l^mpressed into all black regions. 

n AGC amplifier 18 typically has several different gain settings which may be applied to the 
^nsor output signal. The required gain setting for a particular image is selected based on the 
total light entering camera 10. If the total light level is low, the gain setting is increased. 
Similarly, if the total light level is high, the gain setting is decreased. The particular gain setting 
selected is applied uniformly to the entire image. A/D converter 20 generates a digital video 
output signal on a signal line 22. As discussed above, the dynamic range of the digital video 
output signal is limited by the capacity of A/D converter 20. 

Figure 1 also illustrates image processor 12 coupled to camera 10. Image processor 12 is 
an optional component, and is not required for proper operation of camera 10. The digital video 
signal output provided on line 22 may be connected directly to a device capable of accepting 
42390.P3294CR 3 Express Mail No.: EL414969674US 



digital signals. The optional image processor 1 2 includes a video processing circuit 24 and a 
digital-to-analog (D/A) converter 26. Video processing circuit 24 receives the digital video signal 
output fronn line 22 and perfornns various processing of the signal. Various types of video 
processing circuits and video processing functions will be known to those skilled in the art. D/A 
converter 26 receives a signal from the video processing circuit, converts the signal to an analog 
value, and generates an analog video signal output on signal line 28. This analog video signal 
may be transmitted to any device capable of receiving analog video signals. ' 

Problems associated with dynamic range reduction may be solved by utilizing a 10 bit or 
1 2 bit A/D converter to preserve the dynamic range of the sensor output signal. However, 10 
^t and 1 2 bit A/D converters are expensive and substantially increase the cost of the camera. In 
in application requiring an inexpensive camera, the use of 10 bit or 12 bit . A/D converters is 
kpt practical. 

M is therefore desirable to provide a low-cost digital camera utilizing an inexpensive A/D 
gpnverter, yet capable of generating a video signal containing image detail in bright areas and 
iferk areas of the image. 

y H 

SUMMARY OF THE INVENTION 

The present invention provides a digital camera having an inexpensive A/D converter and 
including a processor for enhancing the dynamic range of the camera. The processor instructs a 
gain control amplifier to reduce the gain in bright areas of the image and increase the gain in 
dark areas of the image. These changes in gain settings for different portions of the image 
increase the image detail provided by the camera, thereby improving the overall image quality. 

An embodiment of the present invention provides a sensor for capturing an image and 
generating a sensor output signal. A gain control amplifier is coupled to the sensor and receives 
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the sensor output signal. The gain control amplifier has controls for applying various levels of 
gain to the sensor output signal. An analog-to-digital converter is coupled to the gain control 
amplifier and generates a digital output signal representing the captured image. A processor is 
coupled to the analog-to-digital converter and the gain control amplifier. The processor provides 
a control signal to the gain control amplifier for adjusting the level of gain applied by the 
amplifier. 

Another feature of the present invention provides a gain map containing gain settings 
applied to the sensor output signal by the gain control amplifier. The gain map is continually 
updated by the processor to include changes in the captured image. 

Qther embodiments of the invention provide a register coupled to the processor and the gain 
^ntrol amplifier. The gain map containing gain settings is stored in the register and the gain 
||)ntrol amplifier reads the gain settings from the register. 

if il 

piEF DESCRIPTION OF THE DRAWINGS 

jy The present invention is illustrated by way of example in the following drawings in which 
|j(|e references indicate similar elements. The following drawings disclose various embodiments 
of the present invention for purposes of illustration only and are not intended to limit the scope 
of the invention. 

Figure 1 illustrates a known digital camera and an attached image processor. 

Figure 2 is a first embodiment of a camera and processor according to the present invention. 

Figure 3 is a flow diagram illustrating operation of the present invention. 

Figure 4 is a flow diagram showing the analysis of a captured image and the updating of a gain 
map used by the camera. 

Figure 5 illustrates an exemplary captured image divided into an array of image regions. 

42390.P3294CR 5 Express Mail No.: EL414969674US 



Figures 6A-6C illustrate a gain map at different stages of the image processing procedure. - - 
Figures 7A-7D illustrate examples of histograms generated for various regions of an image. 
Figure 8 is a second embodiment of the invention Including a register within the camera. 
Figure 9 is a third embodiment of the invention having a processor contained within the camera. 
Figure 10 is a fourth embodiment of the invention including additional processing circuitry within 
the camera. 

DETAILED DESCRIPTION 

The following detailed description sets forth numerous specific details to provide a 
giorough understanding of the invention. However, those skilled in the art will appreciate that 
ipe invention may be practiced without these specific details. In other instances, well known 
methods, procedures, components, and circuits have not been described in detail so as not to 
^jbscure the invention. 

Q The present invention provides a system for enhancing the dynamic range of a digital 
gmera having an inexpensive A/D converter. An adaptive gain control mechanism is provided 
TOr adjusting gain settings applied to a captured image, 

A first embodiment of the invention is illustrated in Figure 2. A digital camera 100 
contains a sensor 102 for capturing an image representing a scene 104, Sensor 102 may be any 
type of sensor capable of generating a signal representing a captured image. In a particular 
embodiment of the invention, sensor 102 is a solid state sensor having a dynamic range of 
1000:1 or greater. Sensor 102 generates a sensor output signal representing the captured 
image. The sensor output signal is provided to a gain control amplifier 106 having controls for 
applying various levels of gain to the captured image. In an embodiment of the invention, gain 
control amplifier 106 is an automatic gain control (AGO amplifier capable of receiving and 
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adjusting the sensor intensity at eacli pixel site. An analog output signal from AGC amplifier 106 
is provided to the input of an analog-to-digita! (A/D) converter 108. 

As discussed above, A/D converter 108 may be a relatively inexpensive 6 bit or 8 bit A/D 
converter. To simplify the explanation of the invention, the specification describes camera 100 
and its operation when using an 8 bit A/D converter 108. 

A/D converter 108 converts the signal received from gain control amplifier 106 from an 
analog signal to a digital signal. The digital output, of A/D converter 108 is provided on signal 
line 1 10 and represents an 8 bit digital video signal of the captured image. As noted above, the 
dynamic range of the signal on line 1 10 is limited by the 8 bit capacity of A/D converter 108. 
P The digital video signal on signal line 1 10 is also transmitted to a processor 1 14 using 
^gnal line 1 12. Processor 1 14 may be any type of processor capable of receiving a digital video 
llgnal and performing various operations or calculations related to the received signal. Processor 
ill 4 can be a dedicated image processing system or part of a system servicing other devices or 
gprforming other functions. In a preferred embodiment of the invention, processor 1 14 is a 
pybrsonal computer capable of performing a variety of operations and servicing numerous 
l^vices. In the preferred embodiment, the personal computer utilizes a Pentium@ processor 
manufactured by Intel Corporation of Santa Clara, California. 

The actual operations and processing steps performed by processor 1 14 may be 
implemented in software executed by processor 1 14. Those skilled in the art will appreciate that 
processor 1 14 may be a single processor device capable of processing digital data (e.g., a digital 
signal processor) or processor 1 14 may be a complete computer system containing a variety of 
devices and capable of performing multiple simultaneous operations. Thus, the invention is 
capable of being implemented on a wide variety of processing devices and computer platforms. 
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By way of example, the invention will be described in an implementation utilizing a 
personal computer as processor 1 14. Numerous type of computers may be used to practice the 
invention. The computer must be capable of receiving the digital video signal on signal line 112, 
performing the necessary operations and calculations, and transmitting a control signal to gain 
control amplifier 106 using a communication line 1 16. Line 1 16 functions as a control bus used 
to communicate control signals and data between processor 1 14 and gain control amplifier 106. 
In a specific embodiment of the invention, line 11 6 is a high-speed serial bus such as a Universal 
Serial Bus (USB). Although the invention will be described with reference to a high-speed serial 
bus, any communication line having sufficient bandwidth and low latency may be used to 

rpiplement the invention. 

,■ 

4^ In the embodiment of Figure 2, processor 1 1 4 provides real time control of the gain 
pttings applied by gain control amplifier 106. In this situation, the timely transmission of 
^"pntrol signals is important to proper camera operation. A significant amount of bandwidth and 
fgw latency is required to ensure that gain control amplifier 106 receives the appropriate control 
ijgnals when needed. Alternate embodiments of the invention may require a smaller portion of 
(tie bandwidth and nnay tolerate greater latency. These alternate bandwidth and latency 
requirements will be discussed with respect to the particular embodiments described below. 

Figure 3 is a flow diagram illustrating the operation of the system shown in Figure 2. At 
step 1 18 an image is captured by sensor 102 in camera 100. At step 120, sensor 102 
generates a sensor output signal which is provided to gain control amplifier 106. At step 122, 
gain control amplifier 106 determines appropriate gain settings for various regions of the 
captured image and applies the gain to the corresponding region of the image. The gain settings 
are provided to gain control amplifier 106 by processor 1 14 using line 1 16. Additional details 
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regarding the determination of particular gain settings and dividing the captured image into 
regions are provided below with reference to Figure 4. 

At step 124 of Figure 3, gain control amplifier 106 generates a gain control output signal 
and provides the signal to A/D converter 108. At step 126, A/D converter 108 converts the 
gain control output signal from an analog signal to a digital video output signal. Finally, at step 
128, processor 1 14 analyzes the digital video output signal and updates the gain settings, as 
||eeded, for various regions of the captured image. The steps of Figure 3 are performed 
repeatedly to capture and analyze a series 
of images. 

p Referring to Figure 4, a flow diagram illustrates the analysis of a captured image and the 
#dating of a gain map used by camera 100. One example of a gain map 148 is illustrated in 
|-1gure 6A as a two dimensional array of gain settings. The gain settings may be represented in 
||pcibels (dB) or any other format capable of indicating the gain to be applied by gain control 
Amplifier 106. Each gain setting indicates a specific gain level to be applied by gain control 
ftmplifier 106 to a particular region of the captured image. Gain map 148 includes eight rows 
^d eight columns to create 64 gain map cells 1 50. Figure 6A illustrates a default gain map 
including a zero value stored in each cell 1 50 of the gain map. These default gain settings are 
generated when the system is initialized or reset. During operation of camera 100 and 
associated processor 1 14, the gain settings in the gain map are updated to enhance the 
dynamic range of the digital video output signal, thereby increasing the level of detail provided in 
the digital video signal. In the embodiment of Figure 2, gain map 148 is stored in a register or 
other memory device within processor 1 14. 

Referring again to Figure 4, the procedures illustrated in the flow diagram are performed 
by processor 1 14 (shown in Figure 2), Step 132 initializes a row counter and a column counter 
42390.P3294CR 9 Express Mail No.: EL414969674US 



used to indicate a particular image region and a particular cell of gain map 148 being analyzed or 
updated. At step 134, the captured image is divided into an array of image regions. Figure 5 
illustrates an example of a captured image 1 52 divided into a two dimensional array of image 
regions 154. For purposes of explanation. Figure 5 illustrates the actual scene and does not 
show areas of clipping caused by the A/V converter. 

Image 152 is divided into eight rows and eight columns, creating 64 image regions 154. 
pach image region 1 54 is associated with a particular gain setting contained in a corresponding 
cell of gain map 148 (Figure 6A). Image region (2, 4) is a notation indicating the image region at 
the intersection of row 2 and column 4. The gain level applied to image region (2, 4) is stored in 
^^e gain map at cell (2, 4). Thus, each gain map cell provides a gain level to be applied to the 
i^rresponding image region. Although Figures 5 and 6A illustrate two dimensional arrays having 
^ght rows and eight columns, any number of rows and columns may be used. As the number of 
j^5ws and columns is increased, the size of each image region decreases, resulting in more 
grecise gain adjustments to the captured image. However, additional processing resources are 
inquired to analyze the captured image as the number of rows and columns are increased, 
y At step 136 of Figure 4, the image region indicated by the row and column counters is 
analyzed by processor 1 1 4. Initially, the row and column counters are set to identify image 
region (1, 1). The counters are incremented for each subsequent cycle of the procedure until all 
image regions have been analyzed. Processor 1 14 determines the light level in various portions 
of the image region being analyzed. At step 138, the routine determines whether the image 
region contains significant bright portions; i.e., whether image detail has been lost or "dipped" 
due to the 8 bit limitations of the A/D converter. If the image detail has been dipped at step 
138, then the gain map setting for the image region being analyzed is updated at step 140. Gain 
settings are updated gradually (e.g., 1 dB - 5 dB) until sufficient image detail is provided in each 
42390.P3294CR 10 Express Mail No.: EL414969674US 



region of the captured image. Gain settings are continually updated to compensate for changes 
in the captured image, such as changes in light levels and changes in objects appearing in the 
paptured image. 

If image detail has been clipped from the image region at step 1 38 due to brightness, 
then the gain map setting for that region is reduced at step 140 to provide increased image 
detail; i.e., instead of producing all white regions, the gain setting is reduced to prevent clipping 
nd provide enhanced image detail. 

If significant bright portions are not identified at step 138, then step 142 determines 
whether the image region being analyzed contains significant dark portions; i.e., whether image 
j^tail has been lost or clipped by the 8 bit A/D converter. If image detail has been lost at step 
J42, then the gain map setting for that region is increased at step 140 to provide increased 
Irtiage detail. By increasing the gain setting, previously all black portions of the region may begin 
|§ show some image detail, thereby increasing the overall image quality, 
p If the image region being analyzed does not contain significant bright portions or 
iignificant dark portions, then the routine continues to step 144 without updating the gain map 
jetting for the particular image region. Step 144 determines whether the current row and 
column counters indicate the last row and column of the captured image. If all image regions 
have been analyzed, then the routine ends. Otherwise, the routine branches to step 146 where 
the row and/or column counters are incremented to select the next image region of the captured 
image. The routine then returns to step 136 to analyze the next image region. The image 
regions may be analyzed in any order. Row and column counters represent an example of a 
mechanism for analyzing each image region in a systematic manner. 

When all regions of a captured image have been analyzed according to the procedure 
illustrated in Figure 4, the updated gain map settings are applied to the next captured image by 
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gain control amplifier 106 (Figure 2). The digital video signal generated using the new gain 
settings is then analyzed by processor 1 14 using the procedure shown in Figure 4 to further 
update the gain map settings. Thus, the gain map settings are continually updated to maximize 
the image detail produced in the digital video signal generated by camera 100. 

A particular example of the operations performed in Figure 4 will be described with 
respect to the captured image shown in Figure 5 and the gain maps illustrated in Figures 6A-6C. 
Figure 6A illustrates a default gain map indicating the gain level applied to each region of the 
first captured image. When camera 100 captures the first image, processor 1 14 has no previous 
image data from which to generate gain settings. Therefore, processor 1 14 supplies a default 
gpin map to camera 100 for the first captured image. 

4: After capturing the first image, processor 1 14 divides the image into an array of image 
ggions 154, as shown in Figure 5. Each region 154 is analyzed by processor 1 14 to determine 
Hhether the gain setting for the region should be adjusted to provide increased image detail. 
Msing Figure 5 as the first captured image, certain regions of the image are bright (in the area of 

fr. i: 

ifie window and the sun), certain regions are dark (in the corner of the room to the left of the 
window), and other regions have an average light level (near the desk below the window). 
Processor 1 14 decreases the gain setting associated with bright regions and increases the gain 
setting associated with dark regions. 

Figure 6B illustrates an updated version of gain map 148 after processor 1 14 has 
analyzed the first captured image and adjusted the gain settings accordingly. The gain settings 
illustrated in Figure 6B are then provided to gain control amplifier 106 for adjusting the gain of 
the next captured image. As shown in Figure 6B, the gain settings have been increased along 
the left side of the image (especially in the upper left portion) in an attempt to reduce clipping in 
that area and enhance image details. Gain settings in the bright area of the window in the upper 
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right portion of the image have been reduced to enhance image details. Gain settings around the 
desk (below the window) are unchanged because a sufficient level of detail is already present. 
As discussed above, gain settings are adjusted in small increments (e.g., 1 dB - 5 dB) until 
sufficient image detail is provided in the particular region. This provides a gradual change in the 
image detail, rather than sharp or significant changes between captured images. 

The gain settings illustrated in Figure 68 are applied by camera 100 to the second 
captured image. For this example, the second captured image is the same as Figure 5; i.e., the 
scene being captured has not changed since the first image was captured. Since gain setting 
adjustments are performed gradually, the second captured image requires additional gain setting 
0ijustments to further enhance image details. The digital video output generated by applying the 

5 

^In settings of Figure 68 to the second captured image is provided to processor 1 14 for 
Analysis. As described with respect to the first analysis the second captured image is divided 
||to an array of image regions 1 54. Each image region 1 54 Is analyzed and tiie associated gain 
letting updated to enhance the image details in the region. The results of the second analysis 
are illustrated in Figure 6C. Certain gain settings in the upper left portion of the gain map have 
fe^en increased to provide additional image details in the dark areas. Gain settings in the bright 
window area have been further reduced to enhance image details. The updated gain settings 
illustrated in Figure 6C will be used by camera 1 00 to adjust the gain of the next captured 
image. 

The process of capturing an image, applying settings contained in a gain map to the 
image, and analyzing the digital video output signal to update gain settings is performed 
repeatedly by the system illustrated in Figure 2. The gain settings contained in the gain map are 
updated in response to changes in the captured image; e.g., changes in light level, movement of 
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objects in the image, and the lil<e. Thus, the processor is repeatedly updating the gain map 
settings to enhance the overall image detail provided in the digital video output signal. 

Those skilled in the art will appreciate that various methods may be used by processor 
1 14 to analyze light intensity in each image region. An embodiment of the invention applies a 
histogram equalization algorithm to each image region. The histogram equalization algorithm 
generates a histogram of pixel luminance values in a particular region. The histogram is used to 
identify areas of high luminance and low luminance; i.e., areas requiring gain adjustments. 
Several exemplary histograms are illustrated in Figures 7A-7D. Figures 7A-7D are provided for 
illustration purposes and are not drawn to scale or drawn using the same scale, 
p Figure 7A is a histogram for an image region having a high luminance value. The 
jpstogram illustrates a large number of pixels having a maximum luminance value of 255 (for an 
gbit A/D converter). This histogram may correspond to an image region of Figure 5 in the area 
^ the window. In this situation, histogram equalization is performed to generate an increased 
rgjmber of pixels having values less than 255, thereby enhancing image details in the region. 

ru Figure 7B is a histogram for an image region having a low luminance value, indicated by 

y. 

||ie large number of pixels having a luminance value of zero (the minimum luminance value). This 
histogram may correspond to an image region on the left side of Figure 5. In this situation, 
histogram equalization is performed to generate an increased number of pixels having values 
greater than zero to enhance image details in the region. 

Figure 7C is a histogram for an image region having both low luminance values and high 
luminance values, but few intermediate values. The histogram in Figure 7C identifies a transition 
region; i.e., a transition between a bright area and a dark area. These regions may receive little 
or no gain adjustment because both extremes of brightness and darkness are represented in the 
same image region. If the gain is increased, additional pixels may be increased to a value of 255 
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resulting in additional loss of detail in the bright areas. Similarly, if the gain is reduced, additional 
pixels may be decreased to a value of zero resulting in additional loss of detail in the dark areas. 

Figure 7D is a histogram for an image region having a relatively uniform distribution of 
pixel luminance values throughout the region. This histogram may correspond to an image region 
under the window in Figure 5. In this situation, the histogram is already substantially equalized, 
hnd no gain adjustments are necessary, 

V In addition to the histogram equalization algorithm discussed above, various methods and 
algorithms for smoothing sharp edges and transitions between adjacent image regions will be 
^kfiown to those skilled in the art. These methods and algorithms may be utilized with the 
jp^esent invention to provide smoothing in transition areas and between adjacent image regions. 
^ As noted above, Figure 2 illustrates a first embodiment of the invention including camera 
jjoo and processor 1 14 coupled to the camera. Figures 8-10 illustrate alternate embodiments of 
||»e invention. The operation of these alternate embodiments is similar to the operation described 
Sbove with reference to Figures 3 and 4. 

^' 

y Referring to Figure 8, a second embodiment of the invention is illustrated. A camera 156 

^/ 

includes sensor 102, gain control amplifier 106, and A/D converter 108 as described above with 
respect to Figure 2. Camera 156 generates a digital video signal on line 1 10 which is coupled to 
processor 114 using signal line 112. Processor 1 14 receives the digital video signal, divides the 
signal into a plurality of image regions, and analyzes each region as described above with 
respect to Figures 4, 5, and 6A-6C. Camera 156 also includes a register 160 coupled to 
processor 1 14 using a communication line 158. Register 160 is also coupled to gain control 
amplifier 106 using signal line 162. Register 160 is capable of receiving and storing a gain map 
from processor 1 14 and providing the gain map settings to gain control amplifier 106 as 
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necessary. Register 1 60 may be a relatively small register, capable of storing a single copy of 
the gain map. Gain setting updates and control signals generated by processor 1 14 are 
communicated to register 160 using line 158. Register 160 stores the updated gain settings for 
use by gain control amplifier 106, 

The addition of register 1 60 to camera 1 56 increases the cost of the camera slightly, but 
eliminates the dependence of gain control amplifier 106 on communication line 1 16 {Figure 2) to 
r;.eceive gam settmgs contained in the gain map. If communication line 158 in Figure 8 is shared 
py/ other devices and becomes congested, gain setting updates may not be communicated to 
register 160 in a timely manner. In this situation, gain control amplifier 106 uses the current 
^in settings contained in register 1 60 to adjust the gain of a recently captured image, 
^erefore, this embodiment of the invention may be used with a communication line 1 58 having 
^^smaller available bandwidth and greater latency because the processor is not providing gain 
Jottings to the gain control amplifier in real time. Instead, the gain settings are stored in register 
#^0 and are always available to the gain control amplifier, even if communication line 158 is 
Sbngested. 

y Referring to Figure 9, a third embodiment of the invention is illustrated, A camera 1 64 
includes sensor 102, gain control amplifier 106, and A/D converter 108 as described above with 
respect to Figure 2. A digital video signal is provided on a signal line 166 to a processor 168 
located within camera 164, Processor 168 is coupled to gain control amplifier 106 using line 
170. The operation' of the camera shown in Figure 9 is similar to the operation of the system in 
Figure 2, Camera 1 64 is different from camera 1 00 (Fig ure 2) in that processor 1 68 is contained 
within the camera rather than positioned external to the camera. In this situation, processor 1 68 
is used exclusively by the camera. Therefore, processor 1 68 only requires processing capacity 
to support the operations of the camera. Although the addition of processor 168 within camera 
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1 64 increases the overall cost of the camera, connections associated with an external processor 
(as shown in Figure 2) are eliminated. 

Referring to Figure 10, a fourth embodiment of the invention is shown, A camera 172 
includes sensor 102, gain control amplifier 106, and A/D converter 108 as described above with 
respect to Figure 2. Additionally, camera 172 includes a video processing circuit 180 coupled to 
A/D converter 108 via signal line 174. A digital-to-analog (D/A) converter 182 is coupled to 
pircuit 180 and generates an analog video output signal on output line 184. Processor 1 14 is 
coupled to A/D converter 108 using line 176 and coupled to gain control amplifier 106 using 
communication line 178, The operation of the camera shown in Figure 10 is similar to the 
operation of the system in Figure 2, Camera 172 differs from camera 100 (Figure 2) by 
including circuit 180 and D/A converter 182 within the camera such that the camera produces 
Wi analog video output signal. Video processing circuit 180 may be capable of performing a 
¥priety of different video processing functions, as will be known to those skilled in the art. 
p From the above description and drawings, it will be understood by those skilled in the art 
ittat the particular embodiments shown and described are for purposes of illustration only and 
|tfe not intended to limit the scope of the invention. Those skilled in the art will recognize that 
the invention may be embodied in other specific forms without departing from its spirit or 
essential characteristics. References to details of particular embodiments are not intended to 
limit the scope of the claims, 

CLAIMS 

1 . A camera comprising: 

a sensor configured to capture an image and generate a sensor output signal representing 
the captured image; 
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an amplifier coupled to receive the sensor output signal, wherein the amplifier is 
configured to apply multiple gain levels to the sensor output signal; and 
^ a processor coupled to the amplifier, wherein the processor is configured to provide a 
control signal to the amplifier to adjust the gain levels applied by the amplifier. 

2. The camera of claim 1 wherein the amplifier applies different gain levels to different regions 
bf the captured image. 

3. The camera of claim 1 wherein the processor generates a gain map containing gain settings 
applied to the sensor output signal by the amplifier. 

«,-„"-■: 

If. The camera of claim 3 wherein the gain map is continually updated by the processor to 
lidclude changes in the captured image. 

The camera of claim 3 wherein the gain map is a two dimensional array of gain settings, each 
S^in setting indicating a particular gain applied by the amplifier to a corresponding region of the 
captured image. 

6. The camera of claim 3 further including a register coupled to the processor and the amplifier. 

7, The camera of claim 6 wherein the gain map is stored in the register and the amplifier reads 
the gain settings from the register. 
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8. The camera of claim 1 wherein the processor provides the control signal to the amplifier in 
real-time. 

9. The camera of claim 1 wherein the processor analyzes the sensor output signal to determine 
whether a sufficient level of detail is provided in the sensor output signal. 

10. The camera of claim 9 wherein the processor increases the gain levels in dark portions of 
the captured image and the processor decreases the gain levels in bright portions of the 
captured image. 

,p1. An apparatus for capturing an image, comprising a camera, including: 

a sensor configured to capture the image and generate a sensor output signal representing 
||e captured image; 

p an amplifier coupled to receive the sensor output signal, wherein the amplifier has 
Ilbntrols to apply multiple gain levels to the sensor output signal; and 

y a processor coupled to the camera, wherein the processor is configured to receive the 
sensor output signal, and wherein the processor is configured to provide a control signal to the 
amplifier to adjust the gain level applied by the amplifier, 

12- The apparatus of claim 1 1 wherein the processor generates a gain map containing gain 
settings applied to the sensor output signal by the amplifier. 
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13. The apparatus of claim 12 wherein the gain map is a two dimensional array of gain settings, 
each gain setting indicating a particular gain applied by the amplifier to a region of the captured 
image. 

14. The apparatus of claim 13 wherein the processor divides the captured image into a two 
dimensional array of image regions, each image region associated with a corresponding gain 
setting in the gain map. 

15. The apparatus of claim 12 wherein the camera further includes a register coupled to the 
processor and the amplifier. 

Wp. The apparatus of claim 15 wherein the gain map is stored in the register and the amplifier 
Hads the gain settings from the register. 

fi 

OSS* 

r|J7. A method for enhancing the dynamic range of a sensor output signal representing a 
"piptured image, the method comprising the steps of: 

amplifying the sensor output signal in response to gain settings contained in a gain map, 
wherein each gain setting is associated with a particular region of the captured image; and 

updating the gain settings contained in the gain map in response to changes in the sensor 
output signal. 

18. The method of claim 17 wherein the step of updating the gain settings is performed in 
response to clipping of the amplified sensor output signal. 
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19. The method of claim 17 wherein the step of updating the gain settings includes increasing 
the gain settings in dark portions of the image and reducing the gain settings in bright portions 
of the image. 

20. The method of claim 1 7 wherein the step of updating the gain settings includes dividing the 
captured image into a plurality of image regions, wherein each image region is associated with a 
particular gain setting in the gain map. 

21 . The method of claim 1 7 further including the step of analyzing each image region and 
yDdating the associated gain setting in response to clipping of the amplified sensor output signal 
ijg the image region. 

^J2. An apparatus for capturing an image and generating a signal representing the captured 
hnage, comprising: 

RJ means for amplifying the signal in response to gain settings contained in a gain map, 

P 

wherein each gain setting is associated with a particular region of the captured image; 
means for updating the gain settings contained in the gain map; and 

means for generating a control signal indicating a particular gain setting to be applied to a 
portion of the signal representing the captured image. 

ABSTRACT OF THE DISCLOSURE 

A system for generating a digital output signal representing a captured image includes a sensor 
for capturing the image and generating a sensor output signal. A gain control amplifier is 
coupled to the sensor and receives the sensor output signal. The gain control amplifier has 
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controls for applying various levels of gain to the sensor output signal. An analog-to-digital 
converter is coupled to the gain control amplifier and generates the digital output signal 
representing the captured image, A processor is coupled to the analog-to-digital converter and 
the gain control amplifier. The processor provides a control signal to the gain control amplifier 
for adjusting the level of gain applied by the amplifier. 
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31,195; Jeffrey Sam Smith, Reg. No. 39,377; Maria McComiack Sobrino, Reg. No. 31,639; Stanley W. Sokoloff, 
Reg. No. 25,128; Judith A. Szepesi, Reg. No. 39,393; Vincent P. Tassinari, Reg. No. 42,179; Edwin H. Taylor, Reg. 
No. 25,129; John F. Travis, Reg. No. 43,203; George G. C. Tseng, Reg. No. 41,355; Joseph A. Twarowski, Reg. 
No. 42,191; Lester J. Vincent, Reg. No. 31,460; Glenn E. Von Tersch, Reg. No. 41,364; John Patrick Ward, 
Reg. No. 40,216; Charles T. J. Weigell, Reg. No. 43,398; Kirk D. Williams, Reg. No. 42,229; James M. Wu, Reg. 
No. P45,241; Steven D. Yates, Reg. No. 42,242; Ben J. Yorks, Reg. No. 33,609; and Norman Zafinan, Reg. No. 
26,250; my patent attorneys, and Andrew C. Chen, Reg. No. 43,544; Justin M. Dillon, Reg. No. 42,486; Paramita 
Ghosh, Reg. No. 42,806; and Sang Hui Kim, Reg. No. 40,450; my patent agents, of BLAKELY, SOKOLOFF, 
TAYLOR & ZAFMAN LLP, with offices located at 12400 Wilshire Boulevard, 7th Floor, Los Angeles, California 
90025, telephone (310) 207-3800, and Alan K. Aldous, Reg. No. 31,905; Robert D. Anderson, Reg. No. 33,826; 
Joseph R. Bond, Reg. No. 36,458; Richard C. Calderwood, Reg. No. 35,468; Jeffrey S. Draeger, Reg. No. 41,000; 
Cynthia Thomas Faatz, Reg No. 39,973; Sean Fitzgerald, Reg. No. 32,027; John N Greaves, Reg. No. 40,362; Seth 
Z. Kalson, Reg. No. 40,670; David J. Kaplan, Reg. No. 41,105; Charles A. Mkho, Reg. No. 41,199; Leo V. 
Novakoski, Reg. No. 37,198; Naomi Obinata, Reg. No. 39,320; Thomas C. Reynolds, Reg. No. 32,488; Kenneth M. 
Seddon, Reg. No. 43,105; Mark Seeley, Reg. No. 32,299; Steven P. Skabrat, Reg. No. 36,279; Howard A. Skaist, 
Reg. No. 36,008; Steven C. Stewart, Reg. No. 33,555; Raymond J. Werner, Reg. No. 34,752; Robert G. Winkle, 
Reg. No. 37,474; and Charles K. Young, Reg. No. 39,435; my patent attorneys, and Thomas Raleigh Lane, Reg. No. 
42,781; Calvin E. Wells; Reg. No. P43,256, Peter Lam, Reg. No. P44,855; and Gene I. Su, Reg. No. 45,140; my 
patent agents, of INTEL CORPORATION; and James R. Thein, Reg. No. 31,710, my patent attorney; with frill 
power of substitution and revocation, to prosecute this application and to transact all business in the Patent and 
Trademark Office connected herewith. 

I verily believe the original patent to be wholly or partially inoperative: 

by reason that the patent claims less than I had a right to claim in 
the patent. The claim or claims will be partly inoperative in failing to protect against infringement of all 
embodiments of my invention. Thus, I hereby indicate a desire to seek broadened claims as indicated in the 
Preliminary Amendment filed herewith. I also hereby affirm that this reissue application was filed diligently 
upon discovery of the errors indicated below. The errors arose in the prosecution of the original application 
which resulted in the issuance of the patent. The attorney prosecuting the original application failed to 
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appreciate the scope of the invention, and thus, limited the claims as indicated below. The error arose without 
any deceptive intention on my part. I further acknowledge my duty to disclose information which is material to 
the examination of the application under 37 CFR § 1.56. 

Specifically, in claim 1, the phrases "camera" (column 9, line 58), "a sensor configured to capture an 
image and generate a sensor output signal representing the captured image" (column 9, line 59), "coupled to 
receive the sensor output signal" (column 9, line 61), "configured to apply multiple gain level to the sensor 
output signal" (column 9, line 62), and "configured to provide a control signal to the amplifier" (column 9, line 
65) are individually not necessary to distinguish claim 1 from the prior art or to make claim 1 definite. 

Specifically, in claim 11, the phrases "for capturing an image, comprising a camera" (column 10, line 
27), "a sensor configured to capture the image and generate a sensor output signal representing the captured 
image" (column 10, line 29), "coupled to receive the sensor output signal" (column 10, line 32), "controls to 
apply multiple gain level to the sensor output signal" (column 10, line 33), and "configured to receive the 
sensor output signal, and wherein the processor is configured to provide a control signal to the amplifier" 
(column 10, line 35) are individually not necessary to distinguish claim 1 1 from the prior art or to make claim 
11 definite. 

Specifically, in claim 17, the phrases "for enhancing the dynamic range of a sensor output signal 
representing a captured image" (column 10, line 56), "the sensor output signal in response to gain settings 
contained in a gain map" (column 10, line 59), and "updating the gain settings contained in the gain map in 
response to changes in the sensor output signal" (column 10, line 63) are individually not necessary to 
distinguish claim 17 from the prior art or to make claim 1 1 definite. 
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Title 37, Code of Federal Regulations, Section 1.56 
Duty to Disclose Information Material to Patentability 



(a) A patent by its very nature is affected with a public interest. The public interest is best served, 
and the nnost effective patent examination occurs when, at the time an application is being examined, the ' 
Office is aware of and evaluates the teachings of all information material to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in 
dealing with the Office, which includes a duty to disclose to the Office all information known to that 
individual to be material to patentability as defined in this section. The duty to disclosure information 
exists with respect to each pending claim until the claim is cancelled or withdrawn from consideration or 
the application becomes abandoned. Information material to the patentability of a claim that is cancelled 
or withdrawn from consideration need not be submitted if the information is not material to the patentability 
of any claim remaining under consideration in the application. There is no duty to submit information 
which is not material to the patentability of any existing claim. The duty to disclosure ail information known 
to be material to patentability is deemed to be satisfied if all information known to be material to 
patentability of any claim issued in a patent was cited by the Office or submitted to the Office in the 
manner prescribed by §§1.97(bHd) and 1.98. However, no patent will be granted on an application in 
connection with which fraud on the Office was practiced or attempted or the duty of disclosure was 
violated through bad faith or intentional misconduct. The Office encourages applicants to carefully 
examine: 

(1) Prior art cited in search reports of a foreign patent office in a counterpart application, and 

(2) The closest information over which individuals associated with the filing or prosecution of 
a patent application believe any pending claim patentably defines, to make sure that any material 
information contained therein is disclosed to the Office. 

(b) Under this section, information is material to patentability when it is not cumulative to 
information already of record or being made or record in the application, and 

(1) It establishes, by itself or in combination with other information, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or is inconsistent with, a position the applicant takes in: 

(i) Opposing an argument of unpatentability relied on by the Office, or 

(ii) Asserting an argument of patentability. 

A prima facie case of unpatentability is established when the information compels a conclusion that a 
claim is unpatentable under the preponderance of evidence, burden-of-proof standard, giving each term in 
the claim its broadest reasonable construction consistent with the specification, and before any 
consideration is given to evidence which may be submitted in an attempt to establish a contrary 
conclusion of patentability. 

(c) Individuals associated with the filing or prosecution of a patent application within the 
meaning of this section are: 

(1) Each inventor named in the application; 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively involved in the preparation or prosecution of the 
application and who is associated with the inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. 

(d) Individuals other than the attorney, agent or inventor may comply with this section by 
disclosing information to the attorney, agent, or inventor. 
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